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LECTURE GIVEN AT TH2002: Given a set of Boolean variables, and some constraints between 
them, is it possible to find a configuration of the variables which satisfies all constraints? This 
problem, which is at the heart of combinatorial optimization and computational complexity theory, is 
used as a guide to show the convergence between these fields and the statistical physics of disordered 
systems. New results on satisfiability, both on the theoretical and practical side, can be obtained 
thanks to the use of physics concepts and methods. 

Combinatorial optimization aims at finding, in a finite set of possible configurations, the one which minimizes a 
certain cost function. The famous example of the traveling salesman problem (TSP) can serve as an illustration: A 
salesman must make a tour through N cities, visiting each city only once, and coming back at its starting point. The 
cost function is a symmetric matrix Cy, where Cy is the cost for the travel between cities i and j. A permutation it 
of the N cities gives a tour 7r(l) — > tt(2) — > 7r(3) — > .... Taking into account the equivalence between various starting 
points and the direction of the tour, one sees that the number of distinct tour is (N — l)!/2. For each tour n, the 
total cost is C = c 7r (jv)7r(i) + c 7r(r )7r(r+i) ) which can be computed in N operations. The problem is to find the 

tour 7r with lowest cost. 

As can be seen on this example, the basic ingredients of the optimization problems which will interest us are the 
following: 

• An integer N giving the size of the problem (in the TSP, it is the number of cities) . 

• A set of configurations, which typically scales like exp(iV"). 

• A cost function, which one can compute in polynomial time 0{N 1 ). 
Let me mention a few examples, beside the TSP Jl[. 

In the assignment problem, one is given a set of N persons i — 1,...,N, a set of N tasks a — 1,.,.,N, a N x N 
cost matrix c where a a is the cost for having task a performed by person i. An assignment is a permutation ir G Sn 
assigning task a — ir(i) to person i, and the problem is to find the lowest cost assignment, i.e. the permutation which 

minimizes C = Y^=\ c i-K{i)- 

In the spin glass problem|]2|], one is given a set of N spins, which could be for instance Ising variables <7j G {±1}, 
the total energy of the configuration is a sum of exchange interaction energies between all pairs of spins, E({a}) — 
~J2i<i<j<N Jij (J i <T j^ an d one seeks the ground state of the problem, i.e. the configuration (among the 2 N possible 
ones) which minimizes the energy. 

In physical terms, optimization problems consist in finding ground states. This task can be non trivial if a system 
is frustrated, which means that one cannot get the ground state simply by minimizing the energy locally. This is 
typically what happens in a spin glass. In some sense, statistical physics addresses a more general question. Assuming 
that the system is at thermal equilibrium at temperature T, every configuration C is assigned a Boltzmann probability, 
P(C) = j^e~P E ( c \ Beside finding the ground state, one can ask also interesting questions about which are the typical 
configurations at a given temperature, like counting them (which leads to the introduction of entropy), or trying to 
know if they are located in one single region of phase space, or if they build up well separated clusters, as often 
happens in situation where ergodicity is broken. The generalization introduced by using a finite temperature, beside 
leading to interesting questions, can also be useful for optimization, both from the algorithmic point of view (for 
instance this is the essence of the simulated annealing algorithm||, which is a general purpose heuristic algorithm 
that can be used in many optimization problems), but also from an analytic point of view ||. Conversely, smart 
optimization algorithms turn out to be very useful in the study of frustrated physical systems like spin glasses or 
random field models, and the cross-fertilization between these two fields (and also with the related domain of error 
correcting codes for information transmission |Q|) has been very fruitful in the recent years |Q. 

Before proceeding with one such example, let us briefly mention a few important results in optimization which will 
provide the necessary background and motivation. One of the great achievements of computer science is the theory 
of computational complexity. It is impossible to present it in any details here and I will just sketch a few main ideas, 
the interested reader can study it for instance in 

Within the general framework explained above, we can define three types of optimization problems: the optimization 
problem in which one wants to find the optimal configuration, the evaluation problem in which one wants to compute 



2 



the optimal cost (i.e. the ground state energy), the decision problem in which one wants to know, given a threshold 
cost Co, if there exists a configuration of cost less than Co. 

One classification of decision problems is based on the scaling with N of the computer time required to solve them 
in the worst case. There are two main complexity classes: 

• Class P = polynomial problems: they can be solved in a time t < N a . The assignment is an example of a 
polynomial problem, as is the spin glass problem in 2 dimensions. 

• Class NP = non-deterministic polynomial: Given a 'yes' solution to a NP problem, it can be checked in 
polynomial time. Roughly speaking this means that the energy is computable in polynomial time, so this class 
contains a wide variety of problems, including most of the ones of interest in physics. All problems mentioned 
above are in NP. 

One nice aspect of focusing on polynomiality is that it allows to forget about the details of the definition of N, the 
implementation, language, machine, etc.: any 'reasonable' such change (for instance one could have used the number 
of possible links in the assignment) will change the exponent of N appearing in the computer time of a problem in P, 
but not transform it into an exponential behavior. A problem A is said to be at least as hard as a problem B if there 
exists a polynomial algorithm which transforms any instance of B into an instance of A. 
This allows to define the very important class: 

• NP-complete A problem is NP complete if it is at least as hard as any problem in NP. 

So the NP-complete are the hardest problems in NP. If one such problem can be solved in polynomial time, then all 
the problems in NP are solved in polynomial time. Clearly P is contained in NP, but it is not yet known whether P 
= NP , and this is considered as a major challenge. 

A great result was obtained in this field by Cook in 1971 [Q: The satisfiability problem, which we shall describe 
below, is NP-complete. Since then hundreds of problems have been shown to belong to this class, among which the 
decision TSP or the spin glass in dimension larger or equal to 3. 

The fact that 3-d spin glass is NP-complete while 2-d spin glass is P might induce one to infer that NP-completeness 
is equivalent with the existence of a glass transition. This reasoning is too naive and wrong; the reason is that the 
complexity classification relies on a worth-case analysis, while physicists study the behavior of a typical sample. The 
development of a typical case complexity theory has become a major goal M, also motivated by the experimental 
observations that many instances of NP-complete problems are easy to solve^J . 

One way of addressing this issue of a typical case complexity is to define a probability measure on the instances (= 
the 'samples') of the optimization problem which one is considering. Typical examples are: 

• TSP with independent random points uniformly distributed in the unit square 

• assignment with independent affinities uniformly distributed on [0, 1] 

• CuMn spin glass at one percent Mn 

Once this measure has been defined, one is interested in the properties of the generic sample in the N — > oo limit. In 
most cases, global properties (e.g. extensive thermodynamic quantities, among which the ground state energy), turn 
out to be self averaging. This means for instance that the distribution of the ground state energy density becomes 
more and more peaked around an asymptotic value in the large N limit: almost all samples have the same ground state 
energy density. In this situation, a statistical physics analysis is appropriate. Early examples of the use of statistical 
physics in such a context are the derivation of bounds for the optimal length of a TSP|I, the exact prediction of the 
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ground state energy in the random assignment problem defined above, where the result Eq = 2g-, derived in 1985 
through a replica analysis|fi"o|, was recently confirmed rigorously by Aldous [[ll), or the link between spin glasses and 
graph partitioning [ [lgfl . 

As statistical physics can be quite powerful at understanding the generic structure of an optimization problem, one 
may also hope that it can help finding better optimization algorithms. A successful example which was developed 
recently is the satisfiability problem, to which we now turn. 

As we have seen, satisfiability is a core problem in optimization and complexity theory. It is defined as follows 
[ p"3| : A configuration is a set of N Boolean variables Xi € {0, 1} i = 1, . . . ,N, One is given M constraints which 
are clauses, meaning that they are in the form of an OR function of some variables or their negations. For instance: 
x i V x 2 7 V Xs, in V x 2 , are clauses (notation: x 3 is the negation of x 3 ). So the clause X\ V x 2 7 V x 3 is satisfied if 
either X\ = 1, or x 2 7 = 1, or x 3 = (these events do not exclude each other). The satisfiability problem is a decision 
problem. It asks whether there exists a choice of the Boolean variables such that all constraints are satisfied (we 
will call it a SAT configuration). This is a very generic problem, because one can see it as finding a configuration of 
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the N Boolean variables which satisfies the logical proposition built from the AND of all the clauses (in our example 
[x\ V 227 V 23) A (in V X2) A . . .), and any logical proposition can be written in such a 'conjunctive normal form' . 

Satisfiability is known to be NP complete if it contains clauses of length > 3, but common sense and experience 
show that the problem can often be easy; for instance if the number of constraints per variable a — jj- is small, the 
problem is often SAT, if it is large, the problem is often UNSAT. 

This behavior can be characterized quantitatively by looking at the typical complexity of the random 3-SAT 
problem, defined as follows. Each clause involves exactly three variables, chosen randomly in {xx, ..,x^}; a variable 
appearing in the clause is negated randomly with probability 1/2. This defines the probability measure on instances 
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The properties of random 3-SAT have been first investigated numerically |14, |lq| , and exhibit a very interesting 
threshold phenomenon at a c ~ 4.26: a randomly chosen sample is generically SAT for a < a c (meaning that it is SAT 
with probability 1 when N — > 00), generically UNSAT for a > a c . The time used by the best available algorithms 
(which have an exponential complexity) to study the problem also displays an interesting behavior: For a well below 
a c , it is easy to find a SAT configuration; for a well above a c , it is relatively easy to find a contradiction in the 
constraints, showing that the problem is UNSAT. The really difficult region is the intermediate one where a ~ a c , 
where the computer time requested to solve the problem is much larger and increases very fast with system size. A lot 
of important work has been done on this problem, to establish the existence of a threshold phenomenon, give upper 
and lower bounds on a c , show the existence of finite size effects around a ~ a c with scaling exponents. We refer 
the reader to the literature jlj, [l5|, [l7[ |2(|, and just here extract a few crucial aspects for our discussion. 

The threshold phenomenon is a phase transition, and the neighborhood of the transition is the region where the 
algorithmic problem becomes really hard. 

This relationship between phase transition and complexity makes a statistical physics analysis of this problem 
particularly interesting. Monasson and Zecchina have been the first ones to recognize this importance and to use 
statistical physics methods for an analytic study of the random 3-SAT problem J2~j] , |22] |. Basically this problem falls 
into the broad category of spin glass physics. This is immediately seen through the following formulation. To make 
things look more familiar, physicists like to introduce for each Boolean variable Xi £ {0, 1} an Ising spin <t, £ { — 1,1} 
through Xi — -^r* . A satisfiability problem like 



(21 V 2 27 V x 3 )A(xu Vx 3 V 2 2 )A. . . (1) 



can be written in terms of an energy function, where the energy is equal to one for each violated clause. Explicitly, 
in the above example, one would have 

1 + U\ 1 + CT 2 7 1 - 03 1 - (Tn 1 + CT 3 1 + (72 , ns 

hj = + + . . . iz) 

2 2 2 2 2 2 w 

This is clearly a problem of interacting spins, with 1,2, and 3 spin interactions, disorder (in the choice of which variable 
interacts with which), and frustration (because some constraints are antagonist). More technically, the problem has 
a special type of three-spin interactions on a random hyper-graph. 

Using the replica method, Monasson and Zecchina first showed the existence of a phase transition within the replica 
symmetric approximation, at a c = 5.18, then showed that replica symmetry must be broken in this problem. Some 
variational approximation to describe the replica symmetry breakin g ef fects were developed in particular in [^3[ [24) . 

Recently, in a collaboration with G.Parisi and R. Zecchina |25|, [26[, we have developed a new approach to the 
statistical physics of the random 3-SAT problem using the cavity method. While the cavity method had been 
originally invented to deal with the SK model where the interactions are of infinite range , it was later adapted to 
problems with 'finite connectivity', in which a given variable interacts with a finite set of other variables. While this 
is easily done for systems which are replica symmetric (like the assignment, or the random TSP with independent 
links), it turned out to be considerably more difficult to develop the corresponding formalism and turn it into a 
practical method, in the case where replica symmetry is broken. This has been done in the last two years in joint 
works with G.Parisi J28|], and has opened the road to the study of finite connectivity optimization problems with 
replica symmetry breaking like random K-sat. Curiously, while the cavity method is in principle equivalent to the 
replica method (although it proceeds through direct probabilistic analysis instead of using an analytic continuation 
in the number of replicas), it turns out that it is easier to solve this problem with the cavity method. 

The analytic study of [£5[ ||(| for the random 3-SAT problem at zero temperature shows the following phase diagram 
(see fig .[[]): 

• For a < ad = 3.921, the problem is generically SAT; the solution can be found relatively easily, because the 
space of SAT configurations builds up a single big connected cluster. A T — Metropolis algorithm, in which 
one proposes to flip a randomly chosen variable, and accepts the change iff the number of violated constraints 
in the new configuration is less or equal to the old one, is able to find a SAT configuration. We call this the 
EASY-SAT phase 
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Figure 1: A pictorial description of the phase diagram for random 3-SAT obtained in the cavity method: clusters of SAT (in 
green/light gray) or UNSAT (in red/dark grey) configurations. One finds three qualitatively different phases, the EASY-SAT 
phase for a < ad, the HARD-SAT phase for eta < a < a c , the UNSAT phase obtained for a c < a (see text) 



• For ad < a < a c = 4.267, the problem is still generically SAT, but now it becomes very difficult to find a 
solution (we call this the HARD-SAT phase) . The reason is that the configurations of variables which satisfy all 
constraints build up some clusters. Each such cluster, which we call a 'state', is well separated from the other 
clusters (passing from one to the other requires flipping an extensive number of variables) . But there also exist 
many "metastable states": starting from a random configuration, a local descent algorithm will get trapped in 
some cluster of configurations with a given finite energy (they all have the same number of violated clauses, and 
it is impossible to get out of this cluster towards lower energy configurations through any descending sequence of 
one spin flip moves). The number of SAT clusters Af is exponentially large in N, it behaves as M ~ exp(iVE); but 
the number of metastable clusters is also exponentially large with a larger growth rate, behaving like exp(7VS ms ) 
with E ms > E. The most numerous metastable clusters, which have an energy density e t h, will trap all local 
descent algorithms (zero temperature Metropolis of course, but also simulated annealing, unless it is run for an 
exponential time). 

• For a > a c , the problem is typically UNSAT. The ground state energy density eo is positive. Finding a 
configuration with lowest energy is also very difficult because of the proliferation of metastable states. 

A more quantitative description of the thermodynamic quantities in the various phases is shown in fig.|[ The most 
striking result is the existence of an intermediate SAT phase where clustering occurs. A hint of such a behavior had 
been first found in a sophisticated variational one step replica symmetry breaking approximation developed in p^| ; 
however this approximation predicted a second order phase transition (clusters separating continuously), while we 
now think that the transition is discontinuous: an exponentially large number of macroscopically separated clusters 
appears discontinuously at a = ad- Another point which should be noticed is the fact that the complexity, and the 
energy e t h in the HARD-SAT phase are rather small: around e t h ~ 3 10~ 3 violated clause per variable for a = 4.2. 
This shows that until one reaches problems with at least a few thousands variables, one cannot feel the true complexity 
of the problem. This can explain why the existence of the intermediate phase went unnoticed in previous simulations. 

The second type of results found in [^5[ ||{], |2?j] is a new class of algorithms dealing with the many states structure. 
Basically this algorithm amounts to using the cavity equations on one given sample. Originally the cavity method was 
developed to handle a statistical distribution of samples. For instance in the random 3-SAT case its basic strategy 
is to add one extra variable and connect it randomly to a number of new clauses which has the correct statistical 
distribution. In the large N limit, the statistics of the local field on the new variable should be identical to the 
statistics of the local fields on any other variable in the absence of the new one. It turns out that this strategy can be 
adapted to study a single sample: one considers a given clause a, which involves three variables a\, 02, 03. The cavity 
field on o\ is the field felt by a\ in the absence of a. In the case where there exist many states, there is one such field 
for each possible state of the system, and the order parameter is the survey of these fields, in all the states of fixed 
energy density e: 



PS(h) = C £ <5 (K -V S (^- e ) ( 3 ) 

One can then write a recursion recursion between these surveys. Looking for instance at the structure of figj3[ one 
gets the following iteration equation: 
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Figure 2: Thermodynamic quantities for the random 3-SAT problem: eo is the ground state energy density (minimal number 
of violated clauses per variable); eth is the energy density of the most numerous metastable states, which trap the local descent 
algorithms; £ is the complexity of SAT states with E = 




Figure 3: The basic iterative structure of survey propagation. This subgraph is a part of a 3-SAT problem, in which the 
variable tro belongs to three 3-clauses, involving respectively the variables {ai, n}, {<72, T2}, {0-3, tb}. When the clause 3 is 
switched off, the local cavity field survey Po(h) on spin ao can be computed in terms of the cavity field surveys on each of the 

Spins <7l,Tl,<T2,T 2 . 



PS(h) = & J P c Md 9l P^{hi)dhi P^ 2 (g 2 )dg 2 P e T2 {h 2 )dh 2 

K h ~ f(9i,hx,g2,h 2 ))exp (-^w(g 1 ,h 1 ,g 2 ,h 2 )\ (4) 

The function / just computes the value of the new cavity field on ao in terms of the four cavity fields gi,hi,g 2 ,h 2 . 
It is easily computed by considering the statistical mechanics problem of the five-spin system {<Jq,(Ji,ti,(j 2 ,t 2 \ and 
summing over the 16 possible states of the spins a\ , T\ , a 2 , t 2 . The function w computes the free energy shift induced 
by the addition of tx to the system with the four spins (Ji,T 1 ,a 2 ,T 2 . The exponential reweighting term in (f|) is the 
crucial piece of survey propagation: it appears because one considers the survey at a fixed energy density e. As the 
number of states at energy E = Ne + 8E increases in exp(ySE), where y = this favors the states with a large 
negative energy shift. 

The usual cavity method for the random 3-SAT problem determines the probability, when a new variable is added 
at random, that its survey P§(h) is a given function P(h): the order parameter is thus a functional, the probability 
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Figure 4: The total complexity (= the In of the number of states) as a function of the total energy (= the number of violated 
clauses) for one given sample with N = 10 4 variables and M = 4.2 10 4 constraints. The top curve is the original complexity. 
The next curves are the various complexities obtained for the decimated samples, plotted here every 200 decimations. One 
sees a global decrease of the number of metastable states, and also a global decrease of the threshold energy. In the end the 
problem has no more metastable states and can be solved by simple algorithms. 

of a probability. Because the fields are distributed on integers, this object is not as terrible as it may look and it is 
possible to solve the equation and compute the 'complexity curve' S(£'), giving the phase diagram described above. 

The algorithm for one given sample basically iterates the survey propagation equation on one given graph. It is 
a message passing algorithm which can be seen as a generalization of the belief propagation algorithm familiar to 
computer scientists|]29|]. The belief propagation is a propagation of local magnetic fields, which is equivalent to using 
a Bethe approximation J|(J. Unfortunately, it does not converge in the hard-SAT region because various subparts of 
the graph tend to settle in distinct states, and there is no way to globally choose a state. In this region, the survey 
propagation, which propagates the information on the whole set of states (in the form of a histogram), does converge. 

Based on the surveys, one can detect some strongly biased spins, which are fixed to one in almost all SAT con- 
figurations. The "Survey Inspired Decimation" (SID) algorithm fixes the spin which is most biased, then it re-runs 
the survey propagation on the smaller sample so obtained, and then iterates... An example of the evolution of the 
complexity as a function of the decimation is shown in [|. This algorithm has been tested in the hard SAT phase. It 
easily solves the 'large' benchmarks of random 3sat at a — 4.2 with N = 1000,2000 available at It turns out 

to be able to solve typical random 3-SAT problems with up to N — 10 7 at a = 4.2 in a few hours on a PC, which 
makes it much better than available algorithms. The main point is that the set of surveys contains a lot of detailed 
information on a given sample and can probably be used to find many new algorithms, of which SID is one example. 

To summarize, the recent statistical physics approaches to the random 3-SAT problem give the following results: 

• An analytic result for the phase diagram of the generic samples 

• An explanation for the slowdown of algorithms near a c = 4.267: this is due to the existence of a HARD-SAT 
phase at a G [3.921,4.267], with exponentially many metastable states. 

• An algorithm for single sample analysis^]: Survey propagation converges and yields very non trivial information 
on the sample. It can be used for instance to decimate the problem and get an efficient SAT-solver in the hard- 
SAT region. 

This whole set of results calls for a lot of developments in many directions. 

On the analytical side, the cavity method results quoted above are believed to be correct, but they are not proven 
rigorously. It would be very interesting to turn these computations into a rigorous proof. A very interesting step 
in this direction was taken recently by Franz and Leone who showed that the result for the critical threshold a c 
obtained by the cavity method on random K-SAT with K even actually give a rigorous upper bound to the critical 
a c j32j . The whole construction of the cavity method with the clustering structure, the many states and the resulting 
reweighting, has actually been checked versus rigorous computations on a variant of the SAT problem, the random 
XORSAT problem, where rigorous computations have confirmed the validity of the approach. 

On the numerical side, one needs to develop convergence proofs for survey propagation, and to derive the general- 
ization of the algorithm for the case in which there exists local structures in the interaction graphs. This will amount 
to generalizing from a Bethe like approximation (with many states) to a cluster variational method with larger clusters 
(and with many states). 
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The techniques can also be extended to other optimization problems like coloring [Q. Beside the problems men- 
tioned here, there exist many other fascinating problems of joint interests to physicists and computer scientists, like 
e.g the dynamical behavior of algorithms in optimization or error correcting codes, which I cannot surveyhere. Let 
me just quote a few recent references to help the readers through the corresponding bibliography J3f| ||f],|37], |3g}| . 
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